Article recognition apparatus and article recognition method

ABSTRACT

According to one embodiment, an article recognition apparatus includes an image interface, a weight interface and a processor. The image interface acquires an image captured by photographing a predetermined place where a plurality of articles are disposed. The processor acquires a first image, acquires a second image after detecting a predetermined event, recognizes an article, based on an image of an article area of an article which is absent in the second image, among article areas extracted from the first image, acquires a registered weight of the recognized article from an article database, and outputs an error if total of the registered weights disagrees with a difference weight between a first weight which a weight scale measures at a time of photographing the first image, and a second weight which the weight scale measures at a time of photographing the second image.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.15/425,343, filed Feb. 6, 2017, the entire contents of which areincorporated herein by reference.

FIELD

Embodiments described herein relate generally to an article recognitionapparatus and an article recognition method.

BACKGROUND

There is known an article processing apparatus which recognizesarticles, based on an image captured by photographing articles disposedon a table. The article processing apparatus specifies, from the image,an article area where an article exists, and recognizes the article inthe article area, for example, by scanning a bar code or the like, or byobject recognition.

In related art, there is a problem that the article processing apparatuscannot recognize articles when a plurality of articles are disposed inan overlapping manner.

OBJECT OF INVENTION

In order to solve the above problem, an article recognition apparatusand an article recognition method are provided, which can recognizearticles even when the articles are disposed in an overlapping manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view which schematically illustrates a configuration exampleof a settlement apparatus according to an embodiment.

FIG. 2 is a block diagram illustrating a configuration example of thesettlement apparatus.

FIG. 3 is a view illustrating an example of a first image.

FIG. 4 is a view illustrating a display example of the settlementapparatus.

FIG. 5 is a view illustrating an example of a second image.

FIG. 6 is a view illustrating an example of a commodity area.

FIG. 7 is a view illustrating a display example of the settlementapparatus.

FIG. 8 is a flowchart illustrating an operation example of thesettlement apparatus.

FIG. 9 is a flowchart illustrating an operation example of thesettlement apparatus.

DETAILED DESCRIPTION

In general, according to one embodiment, an article recognitionapparatus includes an image interface, a weight interface and aprocessor. The image interface acquires an image captured byphotographing a predetermined place where a plurality of articles aredisposed. The processor acquires a first image, acquires a second imageafter detecting a predetermined event, recognizes an article, based onan image of an article area of an article which is absent in the secondimage, among article areas extracted from the first image, acquires aregistered weight of the recognized article from an article database,and outputs an error if total of the registered weights disagrees with adifference weight between a first weight which a weight scale measuresat a time of photographing the first image, and a second weight whichthe weight scale measures at a time of photographing the second image.

An embodiment will be described hereinafter with reference to theaccompanying drawings.

To begin with, a settlement according to the embodiment is described.

FIG. 1 schematically illustrates a configuration example of a settlementapparatus 1 according to the embodiment.

The settlement apparatus 1 (article recognition apparatus) executes asettlement process on commodities (articles) in a basket 10. Thesettlement apparatus 1 is installed, for example, in a store which sellscommodities. For example, when the basket 10 was disposed at apredetermined position or when the settlement apparatus 1 accepted apredetermined operation, the settlement apparatus executes a settlementprocess on the commodities in the basket 10. The settlement apparatus 1may be installed as a self-checkout system by which a user performs asettlement process by himself/herself. In addition, the settlementapparatus 1 may be installed as an ordinary cash register by which asalesclerk of the store performs a settlement process.

As illustrated in FIG. 1, the settlement apparatus 1 includes a housing2, a camera 3, a display 4, an operation unit 5, and a weight scale 6.

The housing 2 is a frame which forms the outer shape of the settlementapparatus 1. The housing 2 is formed such that the basket 10 can bedisposed thereon. In the example illustrated in FIG. 1, the housing 2has a square bracket (]) shape, and is formed such that the basket 10can be placed thereon.

The camera 3 photographs commodities in the basket 10. In the exampleillustrated in FIG. 1, the camera 3 is disposed in a manner tophotograph the basket 10 from above. The camera 3 may be disposed in amanner to photograph the inside of the basket 10 obliquely from above.The position and direction for disposition of the camera 3 are notrestricted to a specific configuration.

Incidentally, the settlement apparatus 1 may include a plurality ofcameras 3. In this case, the plural cameras 3 may be disposed in amanner to photograph commodities in the basket 10 at different positionsand angles.

The camera 3 is, for instance, a CCD camera. In addition, the camera 3may be a camera which is configured to photograph invisible light. Thestructure of the camera 3 is not restricted to a specific structure.

The display 4 is a display device which displays an image that aprocessor 21 (to be described later) outputs. The display 4 is, forexample, a liquid crystal monitor.

The user of the settlement apparatus 1 inputs various operationalinstructions to the operation unit 5. The operation unit 5 transmits thedata of the operational instructions, which the operator input, to theprocessor 21. The operation unit 5 is, for instance, a keyboard, numerickeys, and a touch panel. In addition, the operation unit 5 may accept aninput of a gesture from the user.

Here, it is assumed that the operation unit 5 is a touch panel and isformed integral with the display 4.

The weight scale 6 measures the weight of commodities. The weight scale6 is formed on a surface on which the basket 10 is disposed. Forexample, the weight scale 6 measures the weight of the basket 10 and thecommodities in the basket 10. The weight scale 6 transmits the measuredweight to the processor 21. In the meantime, the weight scale 6 maytransmit to the processor 21 a value calculated by subtracting a presetweight of the basket 10.

Incidentally, the camera 3, display 4, operation unit 5 or weight scale6 may be formed integral with the housing 2.

In addition, the settlement apparatus 1 may include, for example, anilluminator which illuminates commodities in the basket 10.

Next, a configuration example of the settlement apparatus 1 isdescribed.

FIG. 2 is a block diagram illustrating a configuration example of thesettlement apparatus 1.

As illustrated in FIG. 2, the settlement apparatus 1 includes the camera3, display 4, operation unit 5, weight scale 6, processor 21, a ROM 22,a RAM 23, an NVM 24, a camera interface 25, a display interface 26, anoperation unit interface 27, and a weight interface 28. The processor21, ROM 22, RAM 23, NVM 24, camera interface 25, display interface 26,operation unit interface 27 and weight interface 28 are interconnectedvia a data bus or the like.

The camera interface 25 and camera 3 are interconnected via a data busor the like. The display interface 26 and display 4 are interconnectedvia a data bus or the like. The operation unit interface 27 andoperation unit 5 are interconnected via a data bus or the like. Theweight interface 28 and weight scale 6 are interconnected via a data busor the like.

Incidentally, the settlement apparatus 1 may include some otherstructure, where necessary, in addition to the structure illustrated inFIG. 2, or may exclude a specific structure.

The camera 3, display 4, operation unit 5 and weight scale 6 are asdescribed above.

The processor 21 includes a function of controlling the operation of theentirety of the settlement apparatus 1. The processor 21 may include aninternal cache and various interfaces. The processor 21 realizes variousprocesses by executing programs prestored in an internal memory or NVM24. The processor 21 is, for example, a CPU.

In the meantime, a part of various functions, which the processor 21realizes by executing the programs, may be realized by hardwarecircuitry. In this case, the processor 21 controls the function which isexecuted by the hardware circuitry.

The ROM 22 is a nonvolatile memory which prestores programs for control,and control data. The ROM 22 is built in the settlement apparatus 1 inthe state in which the ROM 22 stores the control programs and controldata at a stage of manufacture. Specifically, the control programs andcontrol data, which are stored in the ROM 22, are pre-installed inaccordance with the specifications of the settlement apparatus 1.

The RAM 23 is a volatile memory. The RAM 23 temporarily stores, e.g.data which the processor 21 is processing. The RAM 23 stores variousapplication programs, based on instructions from the processor 21. Inaddition, the RAM 23 may store data necessary for the execution of theapplication programs, and execution results of the application programs.

The NVM 24 is composed of, for example, a nonvolatile memory which iscapable of data write and data rewrite, such as an EEPROM (trademark) ora flash ROM. The NVM 24 stores control programs, applications andvarious data in accordance with purposes of operational use of thesettlement apparatus 1. For example, in the NVM 24, program files anddata files are created. Control programs and various data are written inthe respective created files.

The NVM 24 prestores a commodity database. The commodity database storesinformation relating to commodities. The commodity database storescommodity codes and weights of commodities by mutually associating thecommodity codes and weights of commodities. The commodity code isinformation which identifies a commodity. For example, the commoditycode is a numeral, a character, a sign, or a combination thereof. Theweight of a commodity is the weight of a commodity which thecorresponding commodity code indicates.

In the meantime, the commodity database may further includes commoditynames and prices. The structure of the commodity database is not limitedto a specific structure. For example, the processor 21 acquires thecommodity database from an external apparatus, and stores the commoditydatabase in the NVM 24. The NVM 24 may update the commodity database, asneeded, in accordance with a signal from the processor 21.

The camera interface 25 is an interface for communication between theprocessor 21 and camera 3. For example, the processor 21 transmitsthrough the camera interface 25 a signal which causes the camera 3 tocapture an image. In addition, the processor 21 may set cameraparameters for photography in the camera 3 through the camera interface25.

In addition, the camera interface 25 (image interface) acquires an imagewhich the camera 3 photographed. The camera interface 25 transmits theacquired image to the processor 21. The processor 21 acquires the image,which the camera 3 photographed, from the camera interface 25.

The display interface 26 is an interface for communication between theprocessor 21 and display 4. For example, the processor 21 transmits adisplay screen to the display 4 through the display interface 26.

The operation unit interface 27 is an interface for communicationbetween the processor 21 and operation unit 5. For example, theprocessor 21 receives through the operation unit interface 27 a signalwhich indicates an operation that was input to the operation unit 5.

The weight interface 28 is an interface for communication between theprocessor 21 and weight scale 6. For example, the processor 21 receivesa signal, which indicates the weight that the weight scale 6 measured,from the weight scale 6 through the weight interface 28.

Next, the functions which the processor 21 realizes will be described.

To begin with, the processor 21 includes a function of acquiring animage captured by photographing a predetermined place where a pluralityof commodities are disposed. Here, the processor 21 acquires an imagecaptured by photographing a plurality of commodities existing in thebasket 10.

For example, the processor 21 detects that the basket 10 was disposed ina predetermined area. For example, the processor 21 detects that thebasket 10 was disposed on the weight scale 6, based on a signal from theweight scale 6.

Upon detecting that the basket 10 was disposed, the processor 21executes photography of an image (first image) including a plurality ofcommodities existing in the basket 10. For example, the processor 21transmits a signal for photography to the camera 3. The processor 21acquires the first image from the camera 3. In the meantime, in order tophotograph an image, the processor 21 may set photography parameters inthe camera 3.

Incidentally, the processor 21 may acquire the first image from anexternal apparatus.

FIG. 3 illustrates an example of the first image which the camera 3photographed.

In the example of FIG. 3, the first image includes images of commodities31 to 34. As illustrated in FIG. 3, the commodity 31 lies on thecommodity 33 in an overlapping manner. In addition, the commodity 32lies on the commodity 34 in an overlapping manner. The commodities 31and 32 are disposed such that these commodities 31 and 32 can be seenfrom above. Besides, the commodities 33 and 34 are not exposed upward tosuch a degree that these commodities 33 and 34 can be recognized.

The processor 21 also includes a function of acquiring the weight of thecommodities in the basket 10 by using the weight scale 6. For example,the processor 21 receives a signal indicating the weight from the weightscale 6. The processor 21 acquires the weight (first weight) which thesignal indicates. In the example of FIG. 3, the processor 21 acquiresthe weight of the commodities 31 to 34 (or the weight of the commodities31 to 34 and the weight of the basket 10).

Furthermore, the processor 21 includes a function of extractingcommodity areas (article areas), which are image areas of commodities,from the first image.

For example, the processor 21 extracts commodity areas, based on thefirst image. For example, the processor 21 extracts the commodity areasfrom the first image by using edge detection or the like.

In the meantime, the processor 21 may detect the commodity areas fromdistance information which indicates distances from a predeterminedposition to respective parts of the first image. For example, thesettlement apparatus 1 may include a distance sensor.

The method in which the processor 21 extracts the commodity areas is notlimited to a specific method.

The processor 21 also includes a function of presenting the extractedcommodity areas to the user. For example, the processor 21 displays thecommodity areas on the display 4. For example, the processor 21 displaysthe first image. The processor 21 displays the first image incombination with information indicating the commodity areas. Here, it isassumed that the processor 21 displays frames indicating the commodityareas.

FIG. 4 illustrates an example of a display screen which the processor 21displays in order to present the commodity areas. Here, it is assumedthat the processor 21 extracted image areas of the commodities 31 and 32as commodity areas.

As illustrated in FIG. 4, the processor 21 displays a frame 31A and aframe 32A on the first image. The frame 31A indicates the commodity areaof the commodity 31. The frame 32A indicates the commodity area of thecommodity 32.

Incidentally, in this case, the processor 21 may display a guidancewhich prompts take-out of the commodities in the extracted commodityareas from the basket 10. In the example of FIG. 4, the processor 21displays, on the display 4 or the like, a guidance indicating take-outof the commodities in the frame 31A and frame 32A. For example, theprocessor 21 displays a message such as “Please take out commodities,which are surrounded by the frames, from the basket, and put them in adisposable plastic bag or in your shopping bag”.

In addition, the processor 21 includes a function of detecting apredetermined event which indicates completion of take-out ofcommodities in the commodity areas.

For example, the processor 21 accepts, as the predetermined event, anoperation (completion operation) of inputting the completion of take-outof commodities. In the example of FIG. 4, the processor 21 displays anicon 41. The icon 41 is an icon for inputting the completion of take-outof commodities in the commodity areas. For example, the processor 21accepts a touch on the icon 41 as the completion operation.

Incidentally, the processor 21 may accept, as the predetermined event, acompletion operation such as a predetermined gesture.

Additionally, the processor 21 may detect, as the predetermined event,the fact that the weight, which the weight scale 6 measures, does notvary for a fixed time. Besides, the processor 21 may detect thepredetermined event, based on an image photographed by the camera 3. Themethod of detecting the predetermined event is not limited to a specificmethod.

The processor 21 includes a function of acquiring, upon detecting thepredetermined event, an image captured by photographing thepredetermined place once again.

The processor 21 executes, upon detecting the predetermined event,photography of an image (second image) including commodities in thebasket 10 by using the camera 3.

Incidentally, the processor 21 may acquire the second image from anexternal apparatus.

FIG. 5 illustrates an example of the second image which the camera 3photographed. Here, it is assumed that the user took out the commodities31 and 32.

In the example of FIG. 5, the second image includes images of thecommodities 33 and 34. Specifically, in the example of FIG. 5, thecommodities 31 and 32 were taken out, and the commodities 33 and 34,which existed under the commodities 31 and 32, are exposed.

In addition, the processor 21 includes a function of acquiring, upondetecting the predetermined event, the weight of the commodities in thebasket 10 by using the weight scale 6.

The processor 21 acquires the weight (second weight) of the commoditiesin the basket 10 by using the weight scale 6.

The processor 21 also includes a function of generating, based on thefirst image and second image, an image (Region of Interest image (ROIimage)) of the commodity area, which is the extracted commodity area andis the commodity area of the commodity which the user took out (i.e. thecommodity that is absent in the second image).

For example, the processor 21 generates a difference image between thefirst image and second image. For example, the processor 21 generatesthe difference image by subtracting a pixel value of the second imagefrom a pixel value of the first image.

Upon generating the difference image, the processor 21 generates, fromthe difference image, a mask image for extracting the commodity area.For example, the processor 21 generates the mask image by setting thepixel value at 1 if the pixel value of the difference image is not lessthan a predetermined threshold, and by setting the pixel value at 0 ifpixel value of the difference image is less than the predeterminedthreshold. In the mask image, a pixel value at a position where thedifference between the first image and second image is large is 1, and apixel value at a position where the difference between the first imageand second image is small is 0. Specifically, the mask image has a pixelvalue “1” in the commodity area of the commodity which the user tookout, and has a pixel value “0” in other areas.

Upon generating the mask image, the processor 21 generates an ROI image(article area image), based on the first image and the mask image. Forexample, the processor 21 extracts the ROI image by multiplying thepixel value of the first image and the pixel value of the mask image.The ROI image has a pixel value, which is equal to the pixel value ofthe first image, at a pixel position where the difference between thefirst image and second image is not less than the predeterminedthreshold, and has a pixel value “0” at other pixel positions.

FIG. 6 illustrates an example of the ROI image.

ROI images illustrated in FIG. 6 were extracted based on the first imageshown in FIG. 3 and the second image shown in FIG. 5. Accordingly, inthe example of FIG. 6, the ROI images are images of the commodities 31and 32.

In addition, the processor 21 includes a function of recognizing onecommodity or a plurality of commodities, based on the ROI image. Here,it is assumed that the processor 21 acquires, as commodity recognitionresult, a commodity code which indicates a commodity.

For example, the processor 21 reads a bar code in which the commoditycode indicating the commodity is encoded. For example, the processor 21reads the bar code by raster-scanning the ROI image.

Besides, the processor 21 may acquire the commodity code by using objectrecognition. For example, the processor 21 executes object recognition,based on pre-registered dictionaries or commodity images.

In the meantime, the processor 21 may first read the bar code, and then,if the reading of the bar code failed, may execute object recognition.

The method in which the processor 21 recognizes commodities is notlimited to a specific method.

In the example illustrated in FIG. 6, the processor 21 recognizes thecommodities 31 and 32.

The processor includes a function of acquiring a pre-registered weight(registered weight) of a recognized commodity.

For example, the processor 21 refers to a commodity database, andacquires, as the registered weight of the commodity, the weight of thecommodity corresponding to the commodity code of the recognizedcommodity.

Incidentally, the processor 21 may acquire the registered weight of thecommodity from an external apparatus.

In addition, the processor 21 includes a function of calculating adifference weight between the first weight and second weight. Forexample, the processor 21 calculates the difference weight bysubtracting the second weight from the first weight.

In addition, the processor 21 includes a function of determining whetherthe total (total weight) of registered weights of commodities agreeswith the difference weight.

For example, if the difference between the total weight and thedifference weight is not greater than a predetermined threshold, theprocessor 21 determines that both agree. The predetermined threshold maybe preset. Besides, the predetermined threshold may be a value which iscalculated by multiplying the total weight by a predetermined ratio.

In addition, the processor 21 includes a function of determining, if theprocessor 21 determines that the total weight agrees with the differenceweight, whether the basket 10 is empty at the time of photographing thesecond image.

For example, the processor 21 determines whether the basket 10 is emptyor not, based on the weight which the weight scale 6 measures. Forexample, the processor 21 determines that the basket 10 is empty, if theweight of commodities, which the weight scale 6 measures, becomes 0.Besides, the processor 21 may determine whether the basket 10 is emptyor not, based on the second image.

Furthermore, if the processor 21 determined that the basket 10 is notempty, the processor 21 recognizes one again commodities in the basket10. For example, the processor 21 sets the second image and secondweight as the first image and first weight, respectively. For example,the processor 21 overwrites the second image and second weight in thememory that stores the first image and in the memory that stores thefirst weight, respectively.

For example, the processor 21 extracts commodity areas from the firstimage (which was originally the second image). Upon extracting thecommodity areas, the processor 21 displays the extracted commodity areason the display 4.

FIG. 7 illustrates an example of the display screen which the processor21 displays in order to present commodity areas. Here, it is assumedthat the processor 21 extracted the image areas of the commodities 33and 34 as the commodity areas.

As illustrated in FIG. 7, the processor 21 displays a frame 33A and aframe 34A on the first image. The frame 33A indicates the commodity areaof the commodity 33. The frame 34A indicates the commodity area of thecommodity 34. Here, it is assumed that the user takes out thecommodities 33 and 34 from the basket 10, and touches the icon 41.

Upon accepting the touch on the icon 41, the processor 21 photographs asecond image by using the camera 3. Here, it is assumed that the secondimage includes no commodity image. In addition, the processor 21acquires a second weight by using the weight scale 6.

The processor 21 generates ROI images, based on the first image andsecond image. Here, the ROI images are images of the commodities 33 and34.

The processor 21 recognizes the commodities, based on the ROI images.Upon recognizing the commodities, the processor 21 acquires registeredweights of the commodities, and calculates the total weight. Uponcalculating the total weight of commodities, the processor 21 calculatesa difference weight, based on the first weight and second weight. Uponcalculating the difference weight, the processor determines whether thetotal weight agrees with the difference weight. If the processor 21determines that the total weight agrees with the difference weight, theprocessor 21 determines once again whether the basket 10 is empty ornot.

In addition, the processor 21 includes a function of settling therecognized commodities if the processor 21 determines that the basket 10is empty.

For example, the processor 21 refers to the commodity database or thelike, and acquires the prices of the recognized commodities. Theprocessor 21 settles the commodities, based on the total (total amount)of the acquired prices. For example, the processor 21 settles thecommodities by using the user's credit card information or the like. Forexample, the processor 21 may accept an input of credit card informationthrough a card reader from the card that the purchaser possesses.Additionally, the processor 21 may acquire an image of the purchaser byusing a camera, etc., and may acquire credit information correspondingto the image. Additionally, the processor 21 may settle the commoditiesby receiving cash from the user. Additionally, the processor 21 mayacquire SF (Stored Fare) information from the card, and may settle thecommodities, based on the SF information.

In the meantime, the processor 21 may acquire the prices of commoditiesfrom an external apparatus. In addition, the processor 21 may transmitthe total amount to an external apparatus.

The processor 21 also includes a function of outputting, if theprocessor 21 determines that the total weight disagrees with thedifference weight, an error indicating that the commodity recognitionfailed. For example, the processor 21 displays a message prompting analternative action for the user, such as prompting the user to performthe settlement process once again, prompting the user to perform asettlement process at a cash register that is operated by a salesclerk,or prompting the user to call a salesclerk. Incidentally, the processor21 may transmit the error to an external apparatus.

Next, an operation example of the settlement apparatus 1 will bedescribed.

FIG. 8 and FIG. 9 are flowcharts for describing the operation example ofthe settlement apparatus 1.

To start with, the processor 21 of the settlement apparatus 1 determineswhether the basket 10 was disposed on the weight scale 6 (ACT 11). Ifthe processor 21 determines that the basket 10 was not disposed on theweight scale 6 (ACT 11, NO), the processor 21 returns to ACT 11.

If the processor 21 determines that the basket 10 was disposed on theweight scale 6 (ACT 11, YES), the processor 21 acquires a first image byusing the camera 3 (ACT 12). Upon acquiring the first image, theprocessor 21 acquires a first weight by using the weight scale 6 (ACT13).

Upon acquiring the first weight, the processor 21 extracts commodityareas from the first image (ACT 14). Upon extracting the commodityareas, the processor 21 displays the commodity areas on the display 4(ACT 15). Upon displaying the commodity areas, the processor 21determines whether the processor 21 detected a predetermined event (ACT16).

If the processor 21 determines that the processor 21 did not detect thepredetermined event (ACT 16, NO), the processor 21 returns to ACT 16.

If the processor 21 determines that the processor 21 detected thepredetermined event (ACT 16, YES), the processor 21 acquires a secondimage by using the camera 3 (ACT 17). Upon acquiring the second image,the processor 21 acquires a second weight by using the weight scale 6(ACT 18).

Upon acquiring the second weight, the processor 21 generates maskimages, based on the first image and the second image (ACT 19). Upongenerating the mask images, the processor 21 generates ROI images, basedon the first image and the mask images (ACT 20).

Upon generating the ROI images, the processor 21 recognizes commodities,based on the ROI images (ACT 21). Upon recognizing the commodities, theprocessor 21 acquires registered weights of the recognized commodities(ACT 22). Upon acquiring the registered weights, the processor 21calculates a difference weight, based on the first weight and the secondweight (ACT 23).

Upon calculating the difference weight, the processor 21 determineswhether the total of the registered weights agrees with the differenceweight (ACT 24). If the processor 21 determines that the total of theregistered weights agrees with the difference weight (ACT 24, YES), theprocessor 21 determines whether the basket 10 is empty or not (ACT 25).

If the processor 21 determines that the basket 10 is not empty (ACT 25,NO), the processor 21 sets the second image as the first image (ACT 26).Upon setting the second image as the first image, the processor 21 setsthe second weight as the first weight (ACT 27). Upon setting the secondweight as the first weight, the processor 21 returns to ACT 14.

If the processor 21 determines that the basket 10 is empty (ACT 25,YES), the processor 21 settles the recognized commodities (ACT 28).

If the processor 21 determines that the total of the registered weightsdisagrees with the difference weight (ACT 24, NO), the processor 21outputs an error (ACT 29).

When the processor 21 settled the recognized commodities (ACT 28) oroutput the error (ACT 29), the processor 21 terminates the operation.

In the meantime, the processor 21 may not execute the settlement process(ACT 28). The processor 21 may transmit the information (e.g. commoditycodes), which indicate the recognized commodities, to the externalapparatus.

Additionally, the processor 21 may recognize articles other thancommodities. The objects, which the processor 21 recognizes, are notlimited to specific structures.

Additionally, the processor 21 may output an error, when a commodity wastaken out from an area other than the extracted commodity area.

The settlement apparatus with the above-described structure recognizesupper commodities among the commodities which are disposed in anoverlapping manner. In addition, the settlement apparatus prompts theuser to take out the upper commodities which are recognizable, therebycausing lower commodities to be exposed. The settlement apparatusrecognizes the lower commodities. The settlement apparatus continues thesame operation until there remains no commodity. As a result, thesettlement apparatus can effectively recognize the commodities which aredisposed in an overlapping manner.

Additionally, after the user took out commodities, the settlementapparatus executes a recognition process of the taken-out commodities,based on the ROI images that are the images of the taken-outcommodities. Thus, the settlement apparatus can detect the actuallytaken-out commodities among the upper commodities which arerecognizable. As a result, the settlement apparatus can prevent anunlawful act, such as take-out of a non-recognized commodity by theuser, compared to a method in which a commodity, after recognized, istaken out.

Additionally, the settlement apparatus continues the recognition processif the pre-registered weight of the commodity agrees with the weight ofthe taken-out commodity. As a result, the settlement process can preventan unlawful act, such as take-out of a non-recognized commodity by theuser.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

The invention claimed is:
 1. A settlement apparatus for commodities,comprising: an image interface configured to acquire an image capturedby photographing a predetermined place where a plurality of commoditiesare disposed; a weight interface configured to acquire a weight of thecommodities measured by a weight scale, the weight scale beingconfigured to measure the weight of the commodities disposed at thepredetermined place; and a processor configured to: acquire a firstimage through the image interface; acquire a second image through theimage interface after detecting a predetermined event; recognize whichone or more commodities out of the plurality of commodities have beentaken out based on the first and the second image; acquire registeredweights of the recognized one or more commodities from a commoditydatabase; output an error if a total of the registered weights disagreeswith a difference between a first weight which the weight scale measuresat a time of photographing the first image, and a second weight whichthe weight scale measures at a time of photographing the second image;acquire a price corresponding to the recognized one or more commodities;and complete settlement of the commodities based on the acquired pricewhen the weight scale does not detect any commodity.
 2. The settlementapparatus of claim 1, further comprising: a display interface configuredto communicate with a display, wherein the processor is configured tocontrol the display to display commodity areas, which are extracted fromthe first image, on the display through the display interface.
 3. Thesettlement apparatus of claim 2, wherein the processor is configured tocontrol the display to display the first image on the display throughthe display interface, and to display, on the first image, framesindicating the commodity areas extracted from the first image.
 4. Thesettlement apparatus of claim 1, further comprising: an operationinterface, wherein the processor is configured to detect thepredetermined event upon receipt of an input of a predeterminedoperation through the operation interface.
 5. The settlement apparatusof claim 1, wherein the processor is configured to accept apredetermined gesture as the predetermined event.
 6. The settlementapparatus of claim 1, wherein the processor is configured to determine,if the total of the registered weights agrees with the difference,whether one or more commodities are present at the predetermined placeat the time of photographing the second image, if the processordetermines that one or more commodities are present at the predeterminedplace at the time of photographing the second image, set the secondimage as a new first image and set the second weight as a new firstweight, acquire a new second image through the image interface afterdetecting a predetermined event, recognize which one or more commoditieshave been taken out, based on the new first and second images, acquire,from the commodity database, registered weights of the commoditiesrecognized based on the new first and second images, and output an errorif the total of the registered weights disagrees with a differenceweight between the new first weight and a new second weight which theweight scale measures at the time of photographing the new second image.7. The settlement apparatus of claim 1, further comprising: anon-volatile memory storing the commodity database.
 8. The settlementapparatus of claim 1, wherein the processor is configured to generate adifference image between the first image and the second image, togenerate a mask image based on the difference image, and to generate,based on the mask image, an image of commodity areas of the one or morecommodities that have been taken out.
 9. A settlement method ofcommodities, comprising: acquiring a first image captured byphotographing a predetermined place where a plurality of commodities aredisposed; acquiring a second image captured by photographing thepredetermined place after detecting a predetermined event; recognizingwhich one or more commodities out of the plurality of commodities havebeen taken out based on the first and the second image; acquiringregistered weights of the recognized one or more commodities from acommodity database; outputting an error if the total of the registeredweights disagrees with a difference between a first weight which aweight scale measures at a time of photographing the first image, and asecond weight which the weight scale measures at a time of photographingthe second image, the weight scale being configured to measure a weighton the predetermined place; acquiring a price corresponding to therecognized one or more commodities; and completing settlement of thecommodities based on the acquired price when the weight scale does notdetect any commodity.
 10. The settlement method of claim 9, furthercomprising: displaying commodity areas, which are extracted from thefirst image, on a display.
 11. The settlement method of claim 10,further comprising: displaying the first image on the display; anddisplaying, on the first image, frames indicating the commodity areasextracted from the first image.
 12. The settlement method of claim 9,wherein the predetermined event is detected upon receipt of an input ofa predetermined operation through an operation interface.
 13. Thesettlement method of claim 9, further comprising: accepting apredetermined gesture as the predetermined event.
 14. The settlementmethod of claim 9, further comprising: determining, if the total of theregistered weights agrees with the difference, whether one or morecommodities are present at the predetermined place at the time ofphotographing the second image; if one or more commodities are presentat the predetermined place at the time of photographing the secondimage, setting the second image as a new first image and setting thesecond weight as a new first weight; acquiring a new second imagethrough an image interface after detecting a predetermined event,recognizing which one or more commodities have commodity has been takenout based on the new first and second images; acquiring, from thecommodity database, registered weights of the commodities recognizedbased on the new first and second images; and outputting an error if thetotal of the registered weights disagrees with a difference weightbetween the new first weight and a new second weight which the weightscale measures at the time of photographing the new second image. 15.The settlement method of claim 9, further comprising: generating adifference image between the first image and the second image;generating a mask image based on the difference image; and generating,based on the mask image, an image of commodity areas of the one or morecommodities that have been taken out.